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ABSTRACT 

The  Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate  (AFRL/HE)  supports  research  addressing  human 
factors  associated  with  Unmanned  Aerial  Vehicle  (UAV)  operator  control  stations.  Recent  research,  in  collaboration 
with  Rapid  Imaging  Software,  Inc.,  has  focused  on  determining  the  value  of  combining  synthetic  vision  data  with  live 
camera  video  presented  on  a  UAV  control  station  display.  Information  is  constructed  from  databases  (e.g.,  terrain, 
cultural  features,  pre-mission  plan,  etc.),  as  well  as  numerous  information  updates  via  networked  communication  with 
other  sources  (e.g.,  weather,  intel).  This  information  is  overlaid  conformal,  in  real  time,  onto  the  dynamic  camera  video 
image  display  presented  to  operators.  Synthetic  vision  overlay  technology  is  expected  to  improve  operator  situation 
awareness  by  highlighting  key  spatial  information  elements  of  interest  directly  onto  the  video  image,  such  as  threat 
locations,  expected  locations  of  targets,  landmarks,  emergency  airfields,  etc.  Also,  it  may  help  maintain  an  operator’s 
situation  awareness  during  periods  of  video  datalink  degradation/dropout  and  when  operating  in  conditions  of  poor 
visibility.  Additionally,  this  technology  may  serve  as  an  intuitive  means  of  distributed  communications  between 
geographically  separated  users.  This  paper  discusses  the  tailoring  of  synthetic  overlay  technology  for  several  UAV 
applications.  Pertinent  human  factors  issues  are  detailed,  as  well  as  the  usability,  simulation,  and  flight  test  evaluations 
required  to  determine  how  best  to  combine  synthetic  visual  data  with  live  camera  video  presented  on  a  ground  control 
station  display  and  validate  that  a  synthetic  vision  system  is  beneficial  for  UAV  applications. 
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1.  OVERVIEW 

Unmanned  Aerial  Vehicles  (UAVs)  are  aircraft  without  the  onboard  presence  of  a  pilot  or  crew.  Though  the 
physical  separation  of  the  crew  from  the  aircraft  offers  many  promising  benefits,  it  also  presents  challenges  to  the 
effective  design  of  the  UAV  control  station.  Numerous  human  factors  issues  such  as  system  time  delays,  poor  crew 
coordination,  high  workload,  and  reduced  situational  awareness  may  negatively  affect  mission  performance  '.  When 
onboard  an  aircraft,  a  pilot  and  crew  receive  a  rich  supply  of  multi-sensory  information  instantaneously  regarding  their 
surrounding  environment.  UAV  operators,  however,  may  be  limited  to  a  time-delayed,  reduced  stream  of  sensory 
feedback  delivered  almost  exclusively  through  the  visual  channel. 

Of  all  the  information  displays  within  military  UAV  control  stations,  the  video  imagery  from  various  cameras 
mounted  on  the  UAV  is  particularly  valuable.  UAV  pilots  use  imagery  from  the  nose  and  gimbal  cameras  to  verify  clear 
path  for  taxi/runway  operations,  scan  for  other  air  traffic  in  the  area,  and  identify  navigational  landmarks  and  potential 
obstructions.  Additionally,  sensor  operators  use  imagery  from  a  gimbal-mounted  camera  to  conduct  a  wide  variety  of 
intelligence,  surveillance  and  reconnaissance  activities  as  well  as  to  directly  support  combat  operations.  However,  video 
imagery  quality  can  be  compromised  by  narrow  camera  field-of-view,  datalink  degradations,  poor  environmental 
conditions  (e.g.,  dawn/dusk/night,  adverse  weather,  variable  clouds),  bandwidth  limitations,  or  a  highly  cluttered  visual 
scene  (e.g.,  in  urban  areas  or  mountainous  terrain).  If  imagery  interpretation  could  be  enhanced  and  made  more  robust 
under  a  wide  variety  of  situations,  UAV  mission  effectiveness  is  expected  to  increase  substantially. 

Synthetic  vision  systems  can  potentially  ameliorate  negative  video  characteristics  and  enhance  UAV  operator 
interpretation  of  the  imagery.  Spatially-relevant  information  is  constructed  from  databases  (e.g.,  terrain,  cultural 
features,  maps,  etc.)  as  well  as  numerous  real-time  information  updates  via  networked  communication  with 
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other  sources  (e.g.  intelligence  assets,  C2  sources,  etc.)  and  overlaid  conformal  onto  the  dynamic  camera  image  display. 
These  computer-generated  overlays  appear  to  ‘co-exist’  with  real  objects  in  the  imagery,  highlighting  those  points  and 
regions  of  interest  to  operators.  Those  familiar  with  virtual  reality  technology  will  know  of  this  concept  as  ‘augmented 
reality’2.  This  synthetic  vision  overlay  is  hypothesized  to  have  many  benefits.  It  may  improve  operator  situation 
awareness  by  highlighting  information  elements  of  interest  on  the  camera  image,  such  as  threat  location,  the  expected 
location  of  targets,  landmarks,  emergency  airfields,  and  position  of  friendly  forces.  Secondly,  it  may  maintain  the 
operator’s  situation  awareness  of  an  environment  if  the  video  datalink  is  temporarily  degraded  or  lost. 

Synthetic  vision  systems  can  also  serve  to  facilitate  intuitive  networked  communications  between 
geographically  separated  users.  One  concept  is  to  have  ‘on  command’  representation  of  friendly,  neutral,  and  hostile 
forces  using  synthetic  overlays,  allowing  the  UAV  operator  to  look  around  and  “see”  those  around  him/her.  Friendly 
forces,  networked  in  some  manner,  could  share  information  on  their  past  and  present  positions,  as  well  as  planned  paths 
and  possibly  their  action  points,  facilitating  team  interaction.  The  friendly  forces  could  also  pool  their  knowledge  of 
neutral  and  hostile  forces  to  help  maintain  battlespace  awareness.  Conceptually,  the  synthetic  vision  system  can  display 
things  that  cannot  normally  be  seen.  For  example,  perhaps  the  state  of  a  system  can  be  portrayed  based  on  its  emissions 
(e.g.,  radar),  or  machine-to-machine  communications  (e.g.,  data  link  activity)  can  be  highlighted  when  data  is  being 
sent/received.  By  representing  the  activities  and  states,  the  operator  may  be  able  to  gain  additional  situation  awareness 
about  the  surrounding  systems. 

This  paper  describes  an  ongoing  collaboration  between  Rapid  Imaging  Software  Inc.  and  the  Air  Force 
Research  Laboratory’s  (AFRL)  Warfighter  Interface  Division  in  tailoring  and  evaluating  a  synthetic  vision  system  for 
UAV  applications.  Related  human  factors  issues  will  be  delineated  and  discussed.  An  overview  will  then  be  provided 
of  an  AFRL  research  program  that  is  evaluating  the  benefits  of  synthetic  vision  technology  for  UAV  applications  and 
developing  human  factors  guidelines  associated  with  this  technology. 

2.  COLLABORATION  ON  CANDIDATE  UAV  SYNTHETIC  VISION  SYSTEM 

The  Air  Force  Research  Laboratory’s  Warfighter  Interface  Division  has  engaged  in  an  undertaking  to  design 
and  evaluate  the  military  utility  of  conformal  interactive  synthetic  vision  overlay  concepts  tailored  to  UAV  operations. 
Using  detailed  knowledge  of  current  Air  Force  UAV  operations  along  with  established  human  factors  design  practices, 
several  initial  interface  concepts  have  been  generated  and  supporting  hardware  architecture  developed.  In  parallel. 

Rapid  Imaging  Software,  Inc.,  under  a  NASA  research  contract,  has  developed  a  synthetic  vision  product  called  the 
SmartCam3D  System  (SCS)  to  improve  the  situation  awareness  of  NASA  UAV  operators.  The  SCS  has  been 
engineered  to  be  tested  in  operational  UAV  environments  and  has  been  evaluated  by  operators  during  multiple  flights  of 
the  NASA  X-38  UAV  and  the  Army  Shadow  UAV.  The  present  collaboration  brings  these  two  resources  together  to 
design  and  evaluate  tailored  synthetic  visual  overlays  for  various  Air  Force  UAV  and  C2  applications,  including  those 
involving  teleoperated  and  small  UAV  applications. 

2.1  SmartCam3D  Synthetic  Vision  System 

The  SmartCam3D  (SCS)  is  an  enhanced  visualization  technology  developed  by  Rapid  Imaging  Software  Inc.  as 
part  of  a  NASA  X-38  RPV  flight-test  effort.  Subsequently,  it  was  matured  during  an  integration  effort  for  an  Army 
UAV  program.  For  the  present  effort,  it  has  been  tailored  for  Air  Force  UAV  applications  through  collaboration  with  the 
AFRL.  This  system  combines  real-time  synthetic  vision  with  live  video,  in  an  attempt  to  enhance  the  situation 
awareness  of  UAV  operators  across  a  wide  range  of  missions  and  environmental  conditions  (Figure  1).  This  technology 
provides  the  users  with  real-time  video  that  is  enriched  with  conformal  spatially-relevant  scene  information  from 
multiple  sources  (database,  mission  plan,  real-time  intel  updates,  etc.).  The  goal  is  to  effectively  increase  the  signal-to- 
noise  ratio  of  the  imagery,  allowing  operators  to  more  quickly  locate,  identify,  and  act  on  critical  information. 

The  SCS  consists  of  three  physical  components:  the  UAV  platform,  a  camera  which  produces  a  video  image 
stream,  and  a  computer  (PC  or  laptop)  with  the  SCS  software  to  create  a  synthetic  vision  which  matches  the  camera 
(Figure  2).  The  SCS  computer  is  stationed  in  the  control  station  and  has  a  geographic  database  that  the  software  uses  to 
create  the  synthetic  view.  Additional  network  feeds  are  needed  to  provide  SCS  with  real-time  intelligence  and  C2 
updates.  The  notion  of  SCS  is  simple.  A  video  camera  is  mounted  on  the  aircraft  in  such  a  manner  as  to  provide  the 


operator  with  a  view  from  the  vehicle.  At  the  same  time,  a  computer  creates  a  three-dimensional  representation  of  the 
current  scene  that  the  camera  should  be  viewing.  Doing  this  requires  a  camera  bore-sight  calibration  procedure  to  co¬ 
align  the  real  and  simulated  cameras.  This  done,  the  two  streams  of  video  are  overlaid  inside  of  the  computer. 


Figure  1.  SmartCam3D  (SCS)  display  illustrating  spatially  referenced  computer-generated  overlay  symbology 

onto  real-time  video  imagery. 
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Figure  2.  Basic  components  of  the  SmartCam3D  System  (SCS). 


Another  critical  component  for  the  SCS  operation  is  a  real-time  video  stream  and  position/attitude  data  for  the 
aircraft.  The  specific  data  that  is  required  includes:  latitude,  longitude,  altitude,  pitch,  heading  and  roll.  Additionally,  if 
the  camera  is  mounted  on  a  gimbal  and  has  zoom  capability,  then  gimbal  angles  and  zoom  settings  are  also  required  in 
the  data  stream.  Realistic  performance  requires  high  data  rates  (i.e.,  greater  than  once  per  second).  This  data  allows  the 
software  to  synthesize  a  real-time  synthetic  scene  to  match  the  camera’s  view. 

Symbology  on  the  SCS  is  based  on  information  inserted  by  the  computer  that  details  the  location  of  landing 
zones,  no-fly  zones,  runways,  obstructions,  buildings,  topography  and  other  geographic  data.  Anything  with  known 
geographic  coordinates  can  be  included  in  the  scene.  Since  the  synthetic  vision  system  is  based  on  the  VisualFlight 
software  (www.visualflight.com),  which  is  already  compatible  with  most  NIMA  (National  Imagery  &  Mapping  Agency) 
data  formats  (DTED,  ADRG,  CADRG,  CIB,  etc.),  the  necessary  geographic  data  is  readily  available.  This  synthetic 
scene  is  overlaid  on  the  video  in  real  time,  and  matches  the  camera  view.  As  the  vehicle  flies,  operators  can  look  at  the 
live  video  and  see  the  target  or  landing  locations  overlaid  with  the  synthetic  view.  In  cases  of  night,  poor  weather  or 
other  limited  visibility  environments,  the  operators  can  utilize  the  computer-generated  synthetic  camera  imagery 
(essentially  ‘re-creating’  modeled  components  of  the  real  world  scene  via  computer  graphics).  However,  if  video 
imagery  is  available,  it  can  provide  a  view  that  includes  transient  objects  that  are  not  present  in  the  geographic  database. 
Obstructions  and  hazardous  areas  can  be  clearly  marked,  as  can  important  landmarks  and  desired  landing  points. 
Furthermore,  because  the  synthetically  created  objects  are  generated  from  digital  data,  they  are  not  subject  to  the 
limitations  of  visibility  inherent  to  video.  While  darkness,  terrain  occlusion,  smoke,  fog,  icing,  and  haze  all  impact  the 
video,  the  synthetically  generated  scene  remains  unobstructed. 

2.2  Example  Synthetic  Vision  Overlay  Interface  Concepts  for  UAV  Applications 

Candidate  interface  concepts  were  generated  for  teleoperated  UAV  applications  (examples  are  depicted  in 
Figures  3-6).  These  concepts  are  a  result  of  the  AFRL/Rapid  Imaging,  Inc.  collaboration  effort,  along  with  the  results  of 
a  usability  analysis  conducted  with  UAV  operators  (see  section  4.1).  Validation  of  these  concepts  in  high  fidelity 
simulation  and  flight  tests  is  underway  (see  sections  4.2  and  4.3).  Even  though  this  validation  process  has  not  been 
completed,  the  display  concepts  are  introduced  here  to  make  the  follow  on  discussion  of  human  factors  issues  more 
relevant. 


Figure  3.  Synthetic  vision  symbology  added  to  simulated  UAV  gimbal  video  imagery 
(symbology  marking  threat,  landmarks,  areas  of  interest  and  runway). 


Figure  4.  Synthetic  vision  symbology  added  to  simulated  UAV  nose  camera  video  imagery 
(symbology  marking  threats  and  planned  vehicle  pathway). 


Figure  5.  Synthetic  vision  symbology  for  improving  situation  awareness  in  cluttered  urban  environments. 


Figure  6.  Picture-in-picture  concept,  with  real  video  imagery  surrounded  by  synthetic-generated  terrain  imagery.  Affords  virtual 
expansion  of  the  available  sensor  field-of-view  well  beyond  the  physical  limits  of  the  camera. 

3.  HUMAN  FACTORS  ISSUES  WITH  UAV  SYNTHETIC  VISION  SYSTEMS 

Synthetic  vision  systems  consists  of  computer-generated  ‘worlds’  created  solely  from  various  models  and 
databases.  Presenting  computer  based  information  in  a  conformal  manner  with  sensor  imagery  on  a  unified  display  has 
been  demonstrated  in  past  research  to  reduce  scanning  time,  reduce  the  need  to  mentally  integrate  spatial  information 
from  disparate  sources,  and  facilitate  attentional  focus  and  management3.  It  is  thought  that  the  performance  benefit  is  a 
result  of  the  synthetic  vision  system  highlighting  important  information  elements.  Also,  the  system  can  include 
information  that  does  not  have  a  correlate  in  the  actual  sensor  imagery,  such  as  threat  lethality  envelopes.  Past  research 
has  primarily  focused  on  using  synthetic  vision  systems  to  help  piloting  tasks  during  low-level  flights  with  manned 
platforms4.  The  system  can  convey  self-motion  cues  and  depth  cues  without  occluding  the  sensor  image.  Moreover, 
flight  guidance  symbology  can  be  provided  for  reduced  visibility  conditions,  especially  during  terminal  flight  operations 
such  as  landings.  It  is  anticipated  that  a  synthetic  vision  system’s  highlighting  would  have  similar  benefits  for  UAV 
flight  operations.  Additionally,  ground  target  search  and  identification  tasks  associated  with  many  UAV  missions  will 
likely  benefit  from  this  technology.  However,  there  are  numerous  human  factors  issues  (described  below)  to  consider 
for  UAV  applications  of  synthetic  vision  systems. 

3.1  Location  of  synthetic  vision  symbology 

One  major  design  consideration  is  the  location  of  the  synthetic  vision  symbology:  overlaid  on  the  existing 
camera  display,  on  a  head  coupled  head-mounted  display  (HMD),  or  a  separate  dedicated  display  on  the  console. 
Although  head-coupled  applications  afford  the  potential  to  allow  full  360  degree  viewing  of  an  area,  previous  simulation 
research  has  demonstrated  that  use  of  a  HMD  can  be  detrimental  to  many  UAV  sensor  operator  tasks5.  The  use  of 
separate  displays  reduces  clutter  on  the  sensor  imagery,  however  previous  research  has  shown  that  the  scanning  time 
involved  in  using  a  separate  display  is  often  more  costly  than  the  additional  clutter  imposed  in  overlaying  synthetic 
vision  onto  the  existing  sensor  imagery  display6.  Having  the  information  overlaid  conformally  on  the  camera  display 
may  reduce  scan  time,  minimize  division  of  attention,  and  may  improve  information  retrieval,  but  with  the  potential  cost 
of  additional  clutter  and  the  possibility  of  cognitive  tunneling  (explained  further  below).  This  research  issue  also 
involves  determining  how  information  on  the  head-up  displays  should  correspond  to  head-down  displays.  The  concept 
is  to  maintain  visual  momentum  as  the  individual  shifts  attention  from  head-down  to  head-up  or  vice  versa. 


3.2  Presentation  of  synthetic  vision  symbology:  scene-linked  or  not. 

Assuming  that  the  synthetic  vision  symbology  is  to  be  overlaid  on  the  sensor  image  rather  than  presented  on  a 
separate  display,  it  needs  to  be  determined  how  the  symbology  will  be  presented  simultaneously  with  the  sensor  image. 
Specifically,  will  the  symbology  elements  just  be  superimposed  over  the  sensor  image  or  will  elements  also  be  “scene- 
linked”  (referenced  to  the  world)  such  that  they  undergo  the  same  visual  transformation  as  real  objects  depicted  in  the 
imagery?  An  example  of  the  latter  is  a  virtual  “billboard”  growing  larger  as  the  operator  approaches  the  runway  and  any 
pitch  or  yaw  of  the  aircraft  would  be  perceived  incidentally  when  viewing  the  display.  The  choice  of  presentation 
method  may  be  symbology  element  specific.  One  driving  factor  is  the  degree  to  which  the  UAV  display  is  anticipated  to 
be  cluttered  —  if  there  are  numerous  items  in  short  range,  it  may  not  be  feasible  to  have  them  all  increase  in  size  as  the 
operator  closes  in.  However,  scene-linking  symbology  to  elements  in  the  real  image  may  reinforce  other  motion  cues 
and  benefit  information  retrieval  (see  3.6). 

3.3  Optimization  of  the  synthetic  vision  system  symbology 

3.3.1  Individual  synthetic  vision  symbology  elements.  For  every  display  element,  there  are  a  multitude  of 
related  human  factors  issues.  For  instance,  for  each  line  segment  and  icon  used,  what  is  the  ideal  shape,  color, 
brightness,  contrast,  size,  thickness,  style,  etc.?  What  is  the  ideal  font  size,  color,  background,  etc.  for  any  label  used? 

To  what  degree  of  detail  should  the  labels  provide  information?  Should  the  transparency  of  the  symbology  be 
manipulated  such  that  both  the  video  imagery  and  the  synthetic  vision  symbology  are  simultaneously  visible?  Should 
color  and  size  vary  based  upon  visual  conditions?  For  certain  types  of  symbols,  additional  design  questions  arise.  With 
‘pathway-in-the-sky’  formats,  for  example,  the  appropriate  number  of  pathway  segments  needs  to  be  determined,  along 
with  their  spacing.  Finally,  usability  of  the  candidate  symbology  elements  needs  to  be  evaluated.  For  this  testing,  the 
symbology  should  be  tested  with  a  sensor  image  that  replicates  the  anticipated  background,  sensor  view,  clutter,  light 
level,  etc.  that  will  appear  in  operational  applications. 

3.3.2  Terrain  overlay.  There  are  a  variety  of  methods  that  can  be  employed  to  portray  terrain  in  a  synthetic 
vision  display,  including  a  simple  “gridded  overlay”  (rectangular  grids  of  known  size  to  facilitate  depth  perception), 
terrain  texturing  (e.g.,  colors  correspond  to  different  absolute  terrain  elevations),  and  photo-realistic  terrain  imagery  (i.e., 
from  satellite  imagery  data).  Various  maps  and  other  geographically  referenced  overlays  may  also  be  useful  depending 
on  the  task  at  hand  (e.g.,  FalconView;  http://www.falconview.org/overview.htm).  Once  again,  usability  of  the  candidate 
terrain  overlay  needs  to  be  evaluated  in  representative  UAV  environments. 

3.2.3  Picture-in-picture  (PIP)  presentation.  The  symbology  elements  and  terrain  overlay  mentioned  above 
play  a  key  role  in  the  implementation  of  the  PIP  concept.  With  PIP,  the  video  image  is  condensed  such  that  it  only  takes 
up  a  portion  of  the  display  width  so  that  a  synthetic  view  can  be  presented  surrounding  the  video  image,  thereby  virtually 
increasing  the  overall  field-of-view  available  to  the  operator.  In  implementing  the  PIP  concept,  several  unique  design 
questions  arise:  Should  the  location  of  the  sensor  imagery  be  fixed  in  the  center  of  the  display,  or  should  the  operator  be 
able  to  adjust  its  location?  Should  the  operator  be  able  to  pan  the  field-of-view  presented  on  the  PIP?  How  many  size 
ratios  (synthetic  scene/camera  image)  should  be  made  available  to  the  UAV  operator  and  should  the  operator’s  size 
selection  be  continuous  or  discrete,  the  latter  involving  selection  between  pre-established  ratios?  An  ongoing  simulation 
evaluation  (see  section  4.2)  is  addressing  these  issues  along  with  an  even  more  fundamental  design  question:  does  the 
PIP  concept  improve  situation  awareness  and  target  prosecution? 

3.4  Information  clutter 

The  ability  to  provide  a  synthetic  vision  system  overlaid  on  a  sensor  image  can  have  both  a  positive  and 
negative  impact7  8.  Having  all  the  information  on  one  display  can  minimize  scanning  and  the  effort  required  to  access 
and  monitor  all  the  elements.  However,  the  information  clutter  may  inhibit  the  processing  of  the  fine  detail  in  the  sensor 
imagery  because  of  the  inhibitory  effects  of  overlay  clutter.  Moreover,  the  capability  afforded  by  synthetic  vision 
overlays  to  enable  operators  to  ‘see’  data  about  objects  that  are  not  visible  in  the  real  sensor  imagery  can  increase  clutter. 
For  example,  with  this  “Superman  X-ray  Vision”  a  threat  that  is  visually  occluded  behind  a  mountain  might  continue  to 
be  depicted  with  an  overlaid  symbol.  However,  with  this  portrayal,  the  operator  can  lose  occlusion  cues,  which  are 
important  for  perceiving  depth.  Presenting  “occluded”  objects  with  dotted  or  blurred  outlines  might  help  operators  track 


where  elements  are  located  in  the  three-dimensional  world.  Thus,  the  design  of  the  synthetic  vision  symbology  needs  to 
take  into  account  the  potentially  negative  effects  of  information  clutter  by  only  including  elements  that  will  benefit  the 
operators’  situation  awareness  and  performance  and  employing  design  features  that  minimize  clutter  effects  and 
confusion. 

Regardless  of  the  symbology  set,  operators  should  be  provided  with  the  capability  to  declutter  the  synthetic 
vision  symbology.  A  declutter  function  is  already  provided  in  many  UAV  control  stations  to  control  the  degree  to  which 
flight  symbology  is  portrayed.  A  similar  function  can  be  applied  to  the  synthetic  vision  system  whereby  the  operator  can 
control  the  amount  of  synthetic  information  portrayed.  However,  research  is  needed  on  how  best  to  implement 
decluttering  modes  for  different  UAV  applications.  More  ‘global’  levels  of  declutter  may  be  optimal,  whereby  the 
operator  can  systematically  select  and  deselect  classes  of  information.  For  instance,  perhaps  only  threat  information  is 
desired  with  no  other  synthetic  information.  Another  approach  would  allow  operators  to  de-select  individual  symbology 
elements,  for  instance,  those  that  might  be  adjacent  to  a  target  that  the  operator  needs  to  have  an  unobstructed  view. 

3.5  Information  view  management 

Providing  the  operator  with  the  ability  to  declutter  the  synthetic  vision  system  symbology  is  one  method  of 
managing  how  information  is  presented.  However,  there  are  numerous  other  techniques  for  “view  management’’  which 
maintain  visual  constraints  on  the  projections  of  objects  on  the  display9.  With  appropriate  algorithms,  the  system  can 
prevent  objects  from  occluding  each  other,  by  modifying  selected  object  properties  such  as  position,  size,  and 
transparency.  By  making  adjustments  in  the  manner  in  which  synthetic  vision  symbology  is  presented,  problems  with 
different  synthetic  elements  occluding  each  other  can  be  minimized  as  well  as  a  synthetic  element  occluding  a  key 
element  in  the  real  sensor  image.  Likewise,  an  intelligent  system  can  ensure  that  “distant  text”  does  not  become  illegible 
and  labels  are  automatically  reoriented  and  repositioned  based  on  the  operator’s  viewpoint  with  respect  to  the  object. 
Research  is  needed  to  identify  the  algorithms  of  highest  utility  for  the  task  at  hand. 

Besides  managing  the  view  to  optimize  the  visibility  of  the  synthetic  vision  symbology,  an  intelligent  system 
can  highlight  in  some  fashion  when  new  synthetic  elements  appear  that  are  critical  for  operator  attention.  Conversely, 
the  capability  to  retrieve  dated  information  might  be  useful,  for  instance  to  review  past  flight  paths  or  conduct  battle 
damage  assessment.  The  system  can  also  help  the  operator  retain  spatial  context  with  respect  to  the  overall  situation  by 
interpolating  between  old  and  new  viewpoints  over  a  transitional  period  of  a  few  seconds,  slowing  down  the  rate  of 
transition10.  Evaluation  is  required  to  see  if  this  is  a  benefit  to  spatial  awareness,  outweighing  the  costs  of  less 
responsive  direct  camera  control.  Identifying  useful  coding  methods  to  indicate  the  criticality,  urgency,  and  timeliness 
of  information  depicted  by  elements  is  another  research  topic. 

3.6  Effect  of  synthetic  vision  symbology  on  retrieval  of  non-synthetic  information. 

Cognitive  tunneling  can  occur  when  the  operator  becomes  focused  on  an  element  of  the  synthetic  vision 
symbology  (or  objects  to  which  attention  is  directed  by  the  synthetic  symbology)  to  such  an  extent  that  other  important 
objects  or  events  in  the  sensor  imagery  are  not  attended".  In  the  case  of  UAVs,  this  may  result  in  the  operator  not 
detecting  unexpected,  high-interest  targets.  Scene-linking  the  synthetic  vision  symbology  may  reduce  the  incidence  of 
cognitive  tunneling12.  With  scene-linking,  the  augmented  information  is  integrated  into  the  visual  scene,  rather  than 
superimposed.  It  is  thought  that  scene-linking  helps  by  grouping  the  synthetic  information  and  real  sensor  information 
into  one  perceptual  group,  thus  reducing  problems  associated  with  attentional  allocation.  (This  is  based  on  object-based 
models  of  visual  attention  that  postulate  that  complex  scenes  are  parsed  into  groups  of  objects,  with  attention  focused  on 
only  one  object  at  a  time,  with  object  groups  defined  by  contours,  color,  etc.13).  However,  increasing  the  amount  of 
information  presented  via  synthetic  vision  overlays  could  increase  the  risk  of  cognitive  tunneling  by  the  operator. 

Cognitive  tunneling  is  also  an  issue  for  the  PIP  display  concept  for  synthetic  vision  systems.  With  PIP,  real 
video  imagery  is  surrounded  by  a  synthetic  view,  thereby  virtually  increasing  the  field  of  view  visible  to  the  operator. 
There  is  past  research  evaluating  the  use  of  concurrent  exocentric  maps  for  improving  localization  performance.  In  this 
case,  the  embedded  map  was  opaque,  and  the  operator  could  pan  the  insert  to  see  the  view  behind  it14.  It  is  not  clear 
whether  the  PIP  concept  will  constitute  a  different  perceptual  group,  and  thus  promote  cognitive  tunneling.  The  fact  that 
the  surrounding  view  is  an  extension  of  the  scene  depicted  within  the  PIP  and  that  the  PIP’s  transparency  can  be 
manipulated,  may  help  perceptual  grouping  of  the  two  scenes.  Experimental  evaluations  are  underway  to  determine  this. 


3.7  Blending  of  synthetic  vision  display  and  sensor  image. 


One  advantage  of  a  synthetic  vision  system  is  its  potential  to  provide  mission  information  when  video  datalink 
is  degraded  or  the  visibility  is  limited.  At  a  maximum  setting,  the  synthetic  vision  imagery  could  totally  replace  the 
sensor  image,  while  other  settings  could  specify  a  blending  of  the  two  information  sources  by  changing  the  transparency 
of  the  entire  synthetic  vision  symbology  set.  Research  issues  include  determining  suitable  methods  to  invoke  imagery 
blending  (discrete  steps  versus  continuous  control)  and  which  type  of  terrain  overlay  is  most  suitable  for  blending. 

Blending  techniques  may  be  appropriate  for  individual  symbology  elements  as  well.  For  instance,  gradual 
blending  of  the  real  and  synthetic  information  along  the  edges  of  the  object  in  the  sensor  image  may  help  create  a  smooth 
transition  between  the  synthetic  and  real  objects  at  the  points  where  occlusion  occurs  or  there  is  an  error  in  registration. 

3.8  Distributed  network  collaborative  communication  of  synthetic  vision  system  information. 

It  is  plausible  that  a  synthetic  vision  system  can  play  a  key  role  in  supporting  distributive  collaborative 
communication  in  the  net-centric  environment  envisioned  for  the  UAV  domain.  Besides  providing  a  common  operating 
picture  of  available  battlespace  information,  one  individual  could  mark  a  specific  spatially  referenced  point  of  interest  on 
a  work  station,  causing  duplicate  informative  synthetic  symbology  to  appear  on  the  displays  of  other  geographically 
separated  stations  in  the  warfare  network.  Thus,  the  synthetic  vision  system  can  be  applied  both  as  a  display  and  as  a 
control.  To  date,  little  research  has  addressed  the  many  issues  associated  with  implementing  such  a  capability.  For 
instance,  one  question  is  how  best  to  keep  each  network  member  informed  on  the  status  of  a  new  designation  -  its 
source,  status  of  coordination  from  others,  timeliness,  priority,  etc.  How  should  far  off  objects,  beyond  one’s  normal 
line-or-sight  or  off-boresight  be  represented?  What  methods  are  suitable  for  teamwork  and  planning? 

3.9  Reliability  of  information. 

Synthetic  vision  systems  are  based  on  data  drawn  from  one  or  more  data  sources  and  the  reliability,  accuracy, 
and  currency  of  that  information  will  vary.  Additionally,  a  source  may  be  reliable  for  one  type  of  information  but  less 
reliable  for  other  information  types.  It  may  be  useful  for  UAV  operators  to  be  able  to  drill  down  to  obtain  knowledge  of 
the  data  source  for  specific  elements,  to  help  judge  the  veracity  of  the  data.  It  may  also  be  possible  to  implement 
algorithms  that  weight  the  reliability  of  information  and  portray  the  certainly  level  with  some  type  of  coding  method. 

3.10  Adequacy  of  the  performance  of  the  synthetic  vision  system. 

Objects  in  the  synthetic  world  and  real  world  must  be  properly  aligned  (i.e.,  registered)  with  respect  to  each 
other  on  the  display,  or  the  illusion  that  the  two  worlds  coexist  will  be  compromised13.  If  registration  errors  are 
systematic,  operators  might  be  able  to  adapt.  Indeed,  that  is  one  research  question:  How  much  registration  error  is 
tolerable  for  a  UAV  application  before  task  performance  degrades  substantially?  Likewise,  how  much  time  delay  can  an 
operator  tolerate?  The  time  delay  discussed  here  refers  to  the  time  difference  between  the  measurement  of  the  position 
and  orientation  of  the  sensor  viewpoint  to  the  moment  when  the  synthetic  image  corresponding  to  that  position  and 
orientation  appears  in  the  display.  Delays  can  cause  registration  errors  and  reduce  task  performance.  There  are  several 
points  in  the  overall  system  that  contribute  to  both  time  delay  and  registration  error,  as  well  as  make  it  likely  that  the 
problems  will  be  variable.  Perhaps  the  most  detrimental  to  the  performance  of  a  synthetic  vision  system  is  the  update 
rate  and  accuracy  of  the  flight  data.  The  quality  of  the  UAV  positional  data,  for  instance,  is  subject  to  quantization  error, 
random  delays,  and  basic  measurement  error,  besides  problems  introduced  by  the  telemetry  system.  Advances  in 
prediction  algorithm  design  may  help  overcome  the  limitations  of  imprecise  and  tardy  data  input  to  the  synthetic  vision 
system.  Manual  intervention  should  also  be  enabled  whereby  the  operator  can  dynamically  recalibrate  the 
correspondence  of  the  synthetic  and  real  worlds. 

3.10  Operator  control  of  synthetic  vision  system  functions. 

The  preceding  subsections  delineate  many  capabilities  that  could  be  implemented,  along  with  the  synthetic 
vision  system,  to  allow  the  operator  to  modify  the  symbology,  e.g.,  amount  of  symbology  presented,  characteristics  of 
the  picture-in-picture,  and  features  of  the  distributed  communication  system.  For  each  of  these  candidate  functions,  the 


ideal  control  interface  needs  to  be  specified.  The  UAV  operators’  conventional  controllers  (keyboard,  mouse,  bezel 
switches,  and  joysticks  to  control  camera  zoom  and  direction  and  UAV  flight)  need  to  be  examined  as  to  how  best  to 
integrate  these  additional  control  requirements.  At  AFRL,  speech-based  control  is  being  considered  whereby  the 
operator’s  speech  signals  are  used  to  carry  out  preset  activities16 

4,  EVALUATION  OF  SYNTHETIC  VISION  SYSTEM  FOR  UAVS 

The  human  factors  issues  raised  in  Section  3  demonstrate  that  there  are  many  research  questions  relative  to  the 
application  of  synthetic  vision  systems  to  UAVs.  What  isn’t  reflected  in  this  section  is  the  potential  interaction  of 
variables.  A  candidate  symbology  concept  may  only  be  beneficial  if  clutter  level  is  low,  visibility  is  good,  or  there  is 
minimal  image  motion.  Or  individual  symbology  concepts  may  show  a  benefit,  but  when  they  are  implemented  together 
in  a  system,  operator  performance  degrades  due  to  unacceptable  clutter,  etc.  Additionally,  many  human  factors 
guidelines  will  be  application-specific.  Thus,  evaluations  are  needed  that  not  only  focus  on  specific  research  issues,  but 
also  evaluate  the  application  of  a  total  candidate  synthetic  vision  system  in  several  different  UAV  task  environments. 

The  end  goal  is  to  determine  if  the  synthetic  vision  system  will  benefit  UAV  operations  and  result  in  increased  mission 
effectiveness.  This  confirmation  involves  several  different  types  of  evaluations,  described  below,  many  of  which  can  be 
performed  in  parallel. 

4.1  Usability  Evaluations 

Evaluations  employing  usability  engineering  tools  enable  a  rapid  design/evaluation/iteration  cycle  to  identify 
promising  synthetic  vision  system  symbology  concepts  for  UAV  applications.  Such  an  evaluation  was  conducted  as  part 
of  the  AFRL/Rapid  Imaging,  Inc.  collaborative  effort17.  With  this  process,  the  most  promising  candidate  concepts  were 
identified,  taking  into  account  operator  ‘profiles’,  ‘use  case  scenarios’  and  function  requirements.  A  critical  design 
review  of  these  concepts  was  then  conducted  with  UAV  operators,  system  developers,  and  human  factors  engineers, 
using  a  series  of  computer-generated  illustrations  of  how  the  synthetic  vision  concepts  would  be  implemented  in  the 
performance  of  the  use-case  scenario.  This  process  was  found  to  be  very  valuable  in  identifying  the  strengths  and 
weaknesses  of  several  specific  symbology  sets.  The  results  of  this  usability  evaluation  are  now  being  addressed  in  detail 
in  simulation  evaluations. 


4.2  Simulation  Evaluations 

The  UAV  control  station  simulation  facility  at  Wright-Patterson 
Air  Force  Base  (Figure  7)  is  being  used  to  support  a  series  of  evaluations  to 
address  many  of  the  issues  identified  in  Section  3,  utilizing  the  most 
promising  symbology  concepts  identified  in  the  usability  evaluation.  The 
high  fidelity  simulation  environment  allows  for  the  adequate  control  of 
experimental  conditions,  manipulation  of  variables  not  possible  in 
operational  flight  tests,  and  use  of  scripted,  off-nominal  events  that  either 
occur  infrequently  or  are  unsafe  to  test  in  a  real-world  environment. 

Events  such  as  camera  slew  error  can  be  inserted  into  the  simulation  in  a 
manner  that  allows  for  complete  repeatability  across  trials.  Specifically, 
these  evaluations  will  help  identify  optimal  control  techniques  for 
information  display,  resolve  issues  of  display  clutter,  and  identify  those 
concepts  that  result  in  highest  mission  performance.  For  example,  the 
objectives  of  a  study  currently  underway  are  to  determine  whether  use  of 
symbology  flagging  landmark  locations  and  the  picture-in-picture  concept 
will  speed  designation  of  known  targets  without  impacting  the  detection  of 
unexpected  targets. 


Figure  7.  UAV  sensor  operator  control  station. 


4.3  Flight  Test  Evaluations 


Flight  demonstrations  and  flight  tests  are  required  to  demonstrate  that  the  synthetic  vision  symbology  can  be 
successfully  integrated  with  the  UAV  platforms  being  targeted  and  that  the  performance  requirements  (3.10)  can 
adequately  be  met.  Flight  tests  can  also  confirm  results  from  simulation  evaluation  that  indicate  that  the  synthetic  vision 
symbology  improves  operator  performance  and  does  not  negatively  impact  any  operator  tasking.  In  other  words,  flight 
tests  validate  whether  the  synthetic  vision  system  can  be  successfully  utilized  and  is  beneficial  in  the  intended  UAV 
environment. 

An  earlier  version  of  the  SmartCam3D  synthetic  vision  system  was  successfully  demonstrated  on  the  NASA  X- 
38  vehicle  during  flight  testing.  During  one  of  the  flights,  a  control  problem  resulted  in  an  unexpected  180  degree  roll. 
Because  the  synthetic  vision  system  offered  improved  situation  awareness,  operators  watching  the  SmartCam  system 
became  aware  of  the  problem  long  before  the  flight  test  engineers,  who  had  to  glean  something  was  amiss  from  a  display 
of  six  rapidly  changing  numbers.  Results  from  these  flight  tests,  as  well  as  subsequent  integration  and  tests,  provide 
further  support  for  the  utility  of  a  synthetic  vision  system.  For  the  candidate  synthetic  vision  system  resulting  from  this 
AFRL/Rapid  Imaging,  Inc.  collaborative  effort,  planning  for  flight  tests  is  underway.  Additionally,  the  system  has 
already  been  successfully  integrated  in  a  UAV  ground  control  ground  control  station  and  favorable  comments  have  been 
received  from  operators. 

4.3  Summary 

Synthetic  vision  system  technology  promises  to  enhance  situation  awareness  for  UAV  operations,  as  well  as 
decrease  workload,  improve  network  collaborative  communication,  and  minimize  effects  of  video  datalink  degradation. 
Operational  benefits  predicted  include  faster  target  acquisition  and  assessment,  more  targets  serviced,  and  reduced 
potential  for  collateral  damage.  There  are,  however,  numerous  questions  pertaining  to  the  design,  implementation,  and 
integration  of  a  synthetic  vision  system  in  UAV  applications. 
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